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Introduction 

Servicemembers  who  suffer  combat,  training  or  accidental  injuries  that  damage  their  sensory 
capabilities  can  have  great  difficulty  returning  to  productive  lifestyles  once  healed  from  their 
initial  trauma.  Sudden  loss  of  vision  can  overwhelm  a  previously  healthy  individual’s  ability  to 
interact  with  the  world,  adversely  impact  the  individual's  recovery  from  physical  and  emotional 
trauma,  and  preclude  a  return  to  the  community  as  a  productive,  stable  member  of  society.  This 
project  seeks  to  advance  technologies  for  non-invasive  vision  sensory  substitution  and 
augmentation  in  order  to  allow  these  individuals  to  return  to  more  normal  social  interactions. 
This  project  exploits  two  specific  capabilities  of  the  human  central  nervous  system,  namely 
cross-modal  sensory  interactions  and  brain  plasticity,  to  address  the  needs  of  these  injured 
servicemembers.  Perception  takes  place  in  the  brain,  not  at  the  end  organ  (Bach-y-Rita,  1972); 
therefore,  the  brain  can  learn  to  reinterpret  the  meaning  of  signals  from  specific  nerves  (e.g., 
from  tactile  receptors)  given  appropriate  self-generated  feedback.  This  forms  the  basis  for 
interfaces  that  can  non-invasively  and  unobtrusively  use  alternative  sensory  pathways  to 
provide  information.  The  Florida  Institute  for  Human  and  Machine  Cognition  (IHMC)  will  develop 
a  proof  of  concept  prototype  Anthro-Centric  Multisensory  Interface  for  Vision 
Augmentation/Substitution  (ACMI-VAS)  system.  We  envision  that  with  appropriate  development, 
the  ACMI-VAS  concept  could  be  integrated  and  reduced  in  size  to  provide  a  robust  situation 
awareness  (SA)  of  visual  information  during  activities  of  daily  living. 

Body 

Sudden  loss  of  vision  can  overwhelm  a  previously  healthy  individual’s  ability  to  interact 
with  the  world,  adversely  impact  the  warfighter’s  recovery  from  physical  and  emotional  trauma, 
and  preclude  a  return  to  the  community  as  a  productive,  stable  member  of  society.  Traumatic 
brain  injury  (TBI)  can  also  induce  loss  of  one  or  more  sensory  capabilities.  Intermittent  or 
permanent  loss  of  veridical  sensory  information  adversely  affects  SA,  leading  to  an  inability  to 
perceive  and  comprehend  the  meaning  of  elements  in  the  environment  and  to  project  their 
future  states.  Without  accurate  SA,  one’s  ability  to  interact  in  a  dynamically  changing  world 
diminishes  (Endsley,  2000).  The  sequelae  of  TBI  and  somatic  polytrauma  suffered  by  this 
growing  population  may  evolve  over  many  months  after  injury,  manifesting  as  significant  loss  of 
one  or  more  sensory  channels  well  after  a  traumatic  event  (Owens,  2008).  While  current 
technologies  for  noninvasive  sensory  substitution  provide  an  inadequate  replacement  for  lost 
sensory  capabilities,  they  can  augment  residual  sensation.  In  addition,  when  integrated  with 
other  substitution  technologies,  these  technologies  can  improve  SA,  and  therefore  open 
opportunities  to  injured  servicemembers  that  they  might  not  otherwise  investigate.  This  research 
and  development  project  seeks  to  leverage  a  number  of  current  technologies  as  well  as  some 
that  are  under  development  into  a  novel  multi-sensory  vision  augmentation/substitution  interface 
that  will  enable  wounded  servicemembers  to  regain  some  measure  of  normal  visual  interactions 
and  functional  return  to  activities  of  daily  living. 

Sensory  loss  can  be  addressed  by  using  precisely  positioned  large  magnetic  fields 
(Kupers,  et  al.,  2006)  or  surgically  with  implanted  devices  like  cortical  (Dobelle,  2000; 
Fernandez,  et  al.,  2005)  and  end  organ  stimulators  such  as  cochlear  or  retinal  implants  (Zhou  & 
Greenberg,  2005;  Weiland,  Liu  &  Humayun  2005;  Veraart,  et  al.,  2003;  Maynard,  Nordhausen  & 
Normann,  1997;  Rauschecker  &  Shannon,  2002),  but  this  adds  both  surgical  trauma  and  risk  of 
infection  (Reefhuis,  et  al,  2003).  These  devices  require  chronic  indwelling  implants  (Figure  1) 
that,  at  the  current  state  of  the  art  only  provide  minimal  resolution  (e.g.,  to  20/120  equivalent 
vision,  Dobelle,  2000),  and  in  the  case  of  cortical  implants  may  induce  seizures  (Javaheri,  et  al., 
2006;  Kotler,  2002).  While  recent  advances  have  dramatically  increased  resolution  available 
from  retinal  implants  (Zrenner,  et  al.,  2010),  they  still  require  invasive  surgery,  indwelling  foreign 
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bodies  and  an  intact  oculus.  Given  the  rapid  development  of  modern  electronics,  wireless 
communications,  batteries  and  computer  technologies,  it  stands  to  reason  that  implanted 
devices  will  also  continue  to  improve  qualitatively  rapidly  and  patients  who  receive  implants  will 
need  to  seriously  consider  the  risks  and  benefits  of  “upgrading”  their  prostheses  through  more 
surgery.  Servicemembers  blinded  from  battlefield  polytrauma  will  have  likely  endured  multiple 
surgical  procedures  by  the  time  they  have  stabilized  sufficiently  to  consider  a  retinal  or  cortical 
implant  and  may  not  be  candidates  due  to  injury  to  underlying  neural  tissue  (retina),  pathways 
(optic  nerve)  or  cortical  processing  (occipital  lobe).  An  alternative  solution  to  this  form  of  sensory 
replacement,  namely  sensory  substitution,  has  shown  promise  for  blind  individuals  and  can 
tolerate  damage  in  any  of  these  components  that  make  up  the  “retinex”  (Land,  1964)  normally 
required  for  human  perception  of  the  visual  environment. 


Figure  1:  (left)  Photograph  of  the  Model  1  epiretinal  microelectrode  array  (Second  Sight  Medical 
Products,  Inc,  Sylmar,  CA)  prior  to  insertion  into  the  ocular  vitreous  cavity.  ( center )  Retinal  Implant  AG 
(Reutlingen,  Germany)  subretinal  combined  photodiode/electrode  array  with  1500  sensors  and 
electrodes,  (right),  X-ray  of  implanted  cortical  array  (Dobelle,  2000)  positioned  on  the  occipital  cortex. 

Perception  takes  place  in  the  brain,  not  at  the  end  organ  (Bach-y-Rita,  1972);  therefore, 
the  brain  can  learn  to  reinterpret  the  meaning  of  signals  from  specific  nerves  (e.g.,  from  tactile 
receptors)  given  appropriate  self-generated  feedback.  This  forms  the  basis  for  interfaces  that 
can  non-invasively  and  unobtrusively  use  alternative  sensory  pathways  to  provide  information. 
This  means  that  the  information  displayed  does  not  necessarily  need  to  represent  the 
underlying  data  at  high  resolution,  rather  abstract  representations  of  the  sensory  environment 
information  can  provide  sufficient  data  for  operator  decision  making  and  improved  SA  (Raj, 
Kass  &  Perry,  2000).  Such  sensory  substitution  mechanisms  exploit  the  plasticity  inherent  to  the 
brain  and  nervous  system,  supporting  both  long  term  and  short  term  anatomical  and  functional 
remapping  of  sensory  data  (Finkel,  1990;  Walcott  &  Langdon,  2001).  Sensory  substitution  refers 
to  the  remapping  of  sensory  data  from  the  normal  sensory  receptor  field  for  a  particular  type  of 
data  to  other  channels  of  information  perception  (Bach-y-Rita,  Collins,  Saunders,  White  & 
Scadden,  1969).  With  appropriate  feedback,  the  plasticity  of  the  human  brain  allows  individuals 
to  learn  to  perceive  the  substituted  data  with  little  cognitive  effort,  especially  if  training  is 
provided  soon  after  sensory  loss  (Bavelier  &  Neville,  2002).  For  maximal  benefit,  the  interface 
must  also  intelligently  pre-process  the  incoming  data  to  account  for  differences  in  capability 
between  the  alternative  channels  and  the  ones  normally  used  to  perceive  given  sensory  data, 
as  well  as  provide  intuitive  control  and  data  management.  Many  sensory  substitution  devices 
and  approaches  have  been  developed  over  the  past  decades  (Machts,  1920).  Modern  computer 
and  electronic  design,  however,  now  enables  the  development  of  intelligent,  noninvasive 
interfaces  unobtrusive  enough  for  use  in  everyday  activities. 

Neural  projections  from  the  visual,  auditory  and  proprioceptive  sensory  systems  interact 
in  the  brainstem  at  the  superior  colliculus  (Meredith  &  Stein,  1986;  Wallace  &  Stein,  2000)  and 
higher  levels  (Fort,  et  al.,  2002),  where  mechanisms  such  as  non-synaptic  diffusion 
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neurotransmission  (as  well  as  synaptic  connections)  can  cross-modally  engage  different 
sensory  channels  (Bach-y-Rita,  1995).  In  addition,  crossmodal  activity  has  been  demonstrated 
in  the  primary  and  secondary  cortical  processing  areas  for  individuals  with  and  without  sensory 
impairment  (Schroeder  &  Foxe,  2005).  For  example,  deaf  individuals  show  activation  in  the 
auditory  cortex  in  response  to  visual  stimuli  (Finney,  Fine  &  Dobkins,  2001),  blind  individuals 
show  visual  cortex  activity  in  response  to  touch  (Sadato,  et  al.,  1996;  Sadato,  et  al.,  2004; 
Sathian,  2005)  and  vibratory  stimuli  (Burton,  Sinclair  &  McLaren,  2004),  and  auditory  stimuli  can 
elicit  changes  in  somatosensory  cortex  activity  (Foxe,  et  al.,  2000).  The  fact  that  these 
interactions  exist  and  become  more  pronounced  given  sensory  channel  deprivation  (Sathian,  K., 
2005)  supports  the  use  of  sensory  substitution  as  a  potential  solution  for  partial  restoration  of 
lost  sensory  channels. 

With  veridical  multisensory  information,  these  mechanisms  appear  to  enhance 
perception  accuracy  and  reaction  time  (Deiderich,  1995),  as  well  as  modulate  ongoing  cognitive 
processes  (Schroeder  &  Foxe,  2005)  while  improving  workload  performance  and  SA  (Wickens 
&  Holland,  1999).  By  exploiting  this  cortical  crossmodal  integration  (Calvert,  2001),  we  and  other 
researchers  have  shown  that  individuals  with  sensory  loss  due  to  artificial  restrictions,  disease, 
congenital  defect,  or  injury  can  use  sensory  substitution  interfaces  to  exploit  this  inherent 
plasticity  of  the  brain  and  nervous  system  for  both  long  term  and  short  term  anatomical  and 
functional  remapping  of  sensory  data  (Walcott  &  Langdon,  2001;  Ptito,  Moesgaard,  Gjedde  & 
Kupers,  2005;  Kaczmarek,  Bach-y-Rita,  Tompkins  &  Webster,  1985)  and  improvement  in  SA 
(Raj,  Kass,  &  Perry,  2000;  Saunders,  Hill  &  Franklin,  1981).  Recent  brain  imaging  studies  have 
confirmed  crossmodal  modulation  of  activity  across  different  sensory  cortices  that  varies 
depending  on  whether  multiple  sensory  channels  provide  congruent  information  or  incongruent 
information  (Johnson  &  Zatorre,  2005;  Jones  &  Callan,  2003;  Fort,  et  al.,  2002;  Laurienti,  et  al., 
2002).  Because  attentional  resources  or  cognitive  effort  required  to  process  the  sensory 
channel  data  also  manifests  as  intersensory  cortical  activity  modulation  (Loose,  et  al.,  2003; 
Kanwisher  &  Wojciulik,  2000;  van  Wassenhove,  Grant  &  Poeppel,  2005;  Mozolic,  et  al,  2008), 
multisensory  aids  for  the  blind  must  minimize  sensory  display  complexity  to  reduce  cognitive 
demands. 

A  number  of  options  for  visual  sensory  substitution  exist,  however,  the  two  most 
investigated  methods  use  audio  and  tactile  representations,  respectively,  of  visual  information. 
The  Braille  tactile  alphabet  represents  the  most  common  example  of  tactile  substitution  using 
two  columns  of  six  or  eight  raised  bumps  either  embossed  on  paper  or  presented  using  a 
mechanical  device.  While  this  has  worked  well  for  plain  text  transcriptions,  it  becomes  unwieldy 
for  graphical  information  for  technical  information  such  as  mathematical  formulas  (Mogo  & 
Archambault,  2003)  and  the  graphically  rich  visuals  of  magazines,  the  Internet  and  the  natural 
world.  (Boehm,  1986;  Ifukube,  Sasaki  &  Peng,  1991).  An  alternative  technique  promoted  by 
Meijer  (1992)  called  “The  vOICe“  system  converts  images  into  an  audio  time  multiplexed 
frequency  and  amplitude  representation  that  sweeps  across  successive  frames  of  video  to 
encode  spatial  information  into  a  complex  audio  stream  (Amedi,  et  al.,  2005).  The  biggest 
drawback  to  these  systems  related  to  the  amount  of  training  required  to  become  fluent  in  the 
alternative  representation  and  the  fact  that  these  methods  heavily  engage  the  remaining 
senses.  With  Braille,  the  reader’s  hands  are  unable  to  perform  other  activities  of  daily  living 
(such  as  making  a  sandwich).  The  vOICe  system  audio  display  and  computerized  text  to 
speech  screen  readers  such  as  Job  Access  With  Speech  (JAWS®,  Freedom  Scientific,  Inc.,  St. 
Petersberg,  FL)  (Bryant  &  Bryant,  2003)  as  well  as  more  recent  multimodal  computer  interfaces 
for  the  blind  (Yu,  et  al.,  2006)  require  significant  auditory  cognitive  engagement  and  may 
prevent  accurate  sensing  of  events  in  the  ambient  acoustical  environment  (e.g.,  a  ringing 
telephone).  Portable  camera  based  readers  such  as  the  knfbReader  Mobile  (K-NFB  Reading 
Technology,  Inc.,  Newton  Lower  Falls,  MA)  allow  the  blind  to  carry  optical  character  recognition 
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into  the  real  world,  but  camera  resolution  and  lack  of  context  of  the  visual  environment 
(Gaudissart,  et  al.,  2005)  make  such  systems  fairly  cumbersome  to  use  in  practice  (e.g.,  the 
user  does  not  have  a  sense  of  the  size  of  the  type  or  if  there  is  any  text  to  recognize  without 
additional  input  from  a  sighted  individual). 

The  Florida  Institute  for  Human  and  Machine  Cognition  (IHMC)  has,  therefore,  focused 
on  integration  and  development  of  non-invasive  methods  of  presenting  visual  information  to 
blinded  servicemembers  using  sensory  substitution.  By  leveraging  previous  work  for  the  United 
States  Navy  and  the  National  Aeronautics  and  Space  Administration  (NASA)  originally  designed 
to  enhance  SA  for  aviators  and  astronauts,  along  with  modern  tactile  technologies  designed  for 
the  blind,  we  have  developed  prototype  hardware  components  that  could  provide  a  substantial 
level  of  visual  sensory  information.  With  early  versions  of  these  components,  we  have  provided 
partial  demonstrations  to  and  solicited  early  feedback  from  four  recently  blinded  military 
servicemembers.  These  evaluators  suffered  polytraumatic  injuries  that  resulted  in  enucleation  of 
both  eyes  10-48  months  prior  to  participation.  All  had  and  at  least  one  ocular  prosthetic  fitted. 
Two  existing  tactile  interfaces  were  demonstrated,  the  U.  S.  Navy/NASA/IHMC  Tactile  Situation 
Awareness  System  (TSAS)  and  the  video  camera  based  Wicab,  Inc.,  (Middleton,  Wl)  BrainPort® 
Wearable  Aid  for  Vision  Enhancement  (BP-Wave  II).  The  former  provided  a  limited  sense  of 
peripheral  visual  object  detection  using  24  electromechanical  vibro-tactile  transducers  (tactors) 
mounted  in  a  garment  that  creates  a  three-dimensional  tactile  torso  interface  (TTI)  array  on  the 
body.  The  latter  consisted  of  an  18x18  array  of  electrotactile  tactile  transducers  placed  on  the 
user’s  tongue  in  a  two  dimensional  array  (Figure  2). 


Figure  2:  Blind  servicemembers  using  prototype  sensory  substitution  interfaces,  (left)  Using  the  TTI  to 
receive  peripheral  attention  directing  tactile  cues,  (center  left)  Using  the  BP-WAVE  II  to  negotiate  stairs 
without  assistance  and  (center  right)  to  read  text,  (right)  BrainPort®  Intraoral  Device  electrotactile  tongue 
array  (600  tactile  pixels). 

With  these  systems  we  have  demonstrated  awareness  of  nearby  objects  on  the  torso 
and,  through  the  BrainPort®  tongue  array,  identification  of  shapes,  shape  orientations,  reading 
(up  to  4-5  word  sentences),  catching  balls  rolled  across  a  table,  navigating  unfamiliar  office 
spaces,  negotiating  stairs,  identifying  open  parking  stalls  from  ones  with  vehicles,  performing 
standard  visual  acuity  tests  to  20/40  equivalent  using  standard  eye  charts  and  identifying 
features  of  family  members  (Figure  3).  At  most,  each  individual  performed  these  tasks  with  4-6 
hours  of  total  training  time. 
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Figure  3:  Initial  BP-WAVE  II  activities  performed  during  technology  demonstrations  with  recently  blinded 
servicemembers.  left  to  right,  top  to  bottom,  Catching  balls,  navigating  office  spaces,  locating  open 
parking  stalls,  noticing  infant’s  hair,  and  reading  eye  charts  to  20/80  (images  used  with  permission). 


In  addition,  IHMC  has  demonstrated  that  a  direct  connection  between  the  tactile 
interface  and  a  computer  graphical  display  can  provide  a  superior  qualitative  experience  by 
bypassing  the  limitations  of  cameras  (e.g.,  glare,  changing  lighting  conditions,  focus  and 
unintentional  movements  of  the  head).  While  this  direct  connection  removes  the  need  to 
position  the  user  in  front  of  a  computer  (a  useful  capability  for  those  with  orthopedic  or  other 
injuries  that  prevent  prolonged  upright  posture),  we  have  prototyped  two  methods  of  tactile 
computer  interactions  that  mimic  sighted  interactions.  Using  video  oculography,  we  have 
demonstrated  that  blind  individuals  can  use  extraocular  musculature  to  control  insensate  eyes 
(even  prosthetic  ones)  with  sufficient  accuracy  to  pan  and  scan  to  read  across  an  image 
presented  tactually.  Likewise,  head  position  can  control  image  zoom  such  that  a  blind  user  can 
lean  toward  or  away  from  an  object  of  interest  to  zoom  in  or  out.  In  the  current  implementation, 
looking  away  from  the  computer  screen  automatically  switches  to  direct  camera  feed,  which 
allows  the  user  to  look  down  at  the  keyboard  when  typing  or  observe  other  items  in  the 
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environment.  Alternatively,  we  have  also  implemented  a  touch  screen  mechanism  that  allows 
the  user  to  feel  the  pixels  under  his  or  her  fingertip  via  the  tongue  while  dragging  across  the 
screen  (Figure  4).  Both  mechanisms  have  been  mastered  by  participants  in  less  than  30 
minutes,  indicating  that  the  method  does  not  significantly  task  cognitive  resources. 


Figure  4:  Direct  computer  tactile  (BrainPort  )  interface  demonstrations,  left,  participant  prepares  to  use 
eye/head  tracking  apparatus  to  determine  point  of  gaze  despite  the  users’  prosthetic  eyes.  The  software 
determines  if  the  participant  is  looking  at  the  screen  and  returns  a  100-200  pixel  sample  of  the  screen 
image.  The  user  intuitively  controls  zoom  level  by  leaning  either  toward  or  away  from  the  screen.  Looking 
down  automatically  selects  a  gaze  directed  subsample  of  the  video  image  from  the  head  mounted 
camera  (allowing  the  user  to  find  keys  on  the  keyboard),  right,  When  the  user  touches  the  screen,  the 
subsection  of  the  screen  image  underneath  the  fingertip  is  presented  to  the  tongue  instead. 

Two  major  issues  were  noted  with  the  prototype  system  evaluations,  namely  registration 
of  gaze  position  relative  to  the  visual  task  and  perception  of  changes  in  the  environment  outside 
the  field  of  view  of  the  camera.  When  reading,  for  example,  inadvertent  head  and  body 
movements  caused  the  camera  image  to  move  away  from  the  word  or  letter  of  interest. 
Reacquisition  of  the  word  or  letter  required  additional  panning  and  scanning  in  order  to  find  the 
line  of  text  and  the  word  or  letter.  The  lack  of  peripheral  visual  sensation  appeared  to  cause  a 
high  level  of  anxiety  for  recently  blinded  individuals  (who  may  not  have  yet  adapted  to  their 
visual  loss).  Blind  individuals  are  often  startled  when  others  approach  quietly  and  begin  talking 
to  or  make  physical  contact  with  them.  These  issues  warranted  the  integration  of  additional 
displays  and  sensors  to  provide  paracentral  and  peripheral  visual  information  and  a  sense  of 
gaze  location  within  the  broader  visual  context. 

Following  these  initial  technology  demonstrations,  IHMC  aggregated  these  prototypes  in 
order  to  improve  a  user's  tactile  visual  sensory  substitution  perception.  Using  the  BP-WAVE-II 
prototype  and  the  previous  generation  VideoTact,  we  determined  that  tasks  that  required  serial 
visual  scan,  such  as  reading,  visual  acuity  testing,  identifying  shapes  arrange  in  rows  and 
columns,  etc.,  could  be  performed  more  easily  when  using  the  two  tactile  interfaces 
simultaneously  (Figure  5). 
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Figure  5:  Initial  mock  up  of  ACMI-VAS  for  foveal  and  paracentral  vision  substitution.  Image  (upper  right 
inset)  presented  on  the  VideoTact  (upper  left  inset)  is  zoomed  in  less  than  the  BrainPort®  image  zoom 
level  (center  display  of  lower  right  inset)  to  provide  a  slightly  wider  field  of  view.  The  physical  size  of  the 
components  has  been  reduced  significantly  and  control  of  the  combined  technologies  has  also  been 
simplified  using  IHMC's  AMI  architecture. 

IHMC  used  its  existing,  in-house  developed  Adaptive  Multiagent  Integration  (AMI) 
software  architecture,  which  can  easily  connect  components  such  as  displays,  sensors, 
algorithms  and  adaptive  automation  via  a  standardized  Java  (Sun  Microsystems,  Inc,  Santa 
Clara,  CA)  interface.  AMI  associates  explicit  ontological  definitions  with  each  agent  that  allow 
rapid  integration  of  new  components  (as  software  agents)  into  the  architecture  because  the 
system  identifies  and  makes  data  connections  between  agents  automatically,  based  on  data 
types  and  relative  quality  of  similar  data  streams.  The  large  complement  of  devices,  including 
video,  motion  capture,  tactile  and  audio  interfaces,  and  pressure,  orientation  and 
psychophysiologic  sensors,  already  been  integrated  as  software  agents  (Johnson,  et  al.,  2005) 
were  leveraged  and  additional  agents  were  created  as  needed.  The  architecture  is  inherently 
scalable,  using  available  processing  power  and  allows  unlimited  nodes  and  agents  on  wired  or 
wireless  networks.  Communications  between  agents  are  made  peer  to  peer  and  support  high 
data  rates  and  both  secure  and  open  transport  mechanisms.  Sensor  signals  or  processed  data 
can  be  delivered  to  multiple  displays  across  different  modalities  (e.g.,  video,  audio,  vibrotactile, 
electrotactile,  etc.)  simultaneously  or  separately,  enabling  side-by-side  real-time  evaluation  of 
various  sensory  substitution  implementations  (Raj,  et  al.,  2005).  Leveraging  AMI  for  this  project 
enabled  rapid  development  of  the  ACMI-VAS  prototype  by  taking  advantage  of  existing  software 
agents  for  a  number  of  tactile  interfaces  (including  torso,  abdominal  and  tongue  placed 
displays),  various  video  cameras,  peripheral  range  and  bearing  sensing,  as  well  as  head  and 
eye  tracking  (e.g.,  OptiTrack,  NaturalPoint,  Inc.,  Corvallis,  OR;  RK-826y-BPCI,  IScan,  Inc., 
Burlington,  MA). 
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By  leveraging  other  funding  vehicles  following  the  initial  technology  demonstrations 
noted  above,  the  ACMI-VAS  system  benefits  from  advances  in  the  BrainPort®,  the  VideoTact 
and  TTI  (Figure  6).  Notably,  IHMC  has  acquired  two  BrainPort®  VI 00  units  to  replace  the 
prototype  BP-WAVE-II  units.  We  have  collaborated  with  ForeThought  Development,  LLC,  to 
design  a  more  compact  VideoTact  system  that  utilizes  modern  electronics  components. 


Figure  6:  (left)  Current  BrainPort®  VI 00  system  showing  sunglasses  mounted  camera,  handheld 
embedded  processor/user  input  device  and  400  tactor  IOD.  (center)  TTI  with  ACU  mounted  C-2  tactors 
and  (right)  comparison  of  C-2  (gold)  and  C-3  (silver)  tactor  sizes. 


Lastly,  we  have  developed  an  reflected  infrared  sensor  range  and  direction  detection 
system  that  can  detect  objects  in  the  environment  and  represent  their  locations  and  motion 
using  the  TTI  (Figure  7),  which  can  use  the  much  smaller,  lighter  weight  C-3  tactors  versus  the 
larger,  older  C-2  tactors  (Engineering  Acoustics,  Inc.,  Casselberry,  FL). 


Figure  7:  (left)  IHMC's  embedded  microprocessor  controlled  24  channel  reflected  infrared  emission  range 
and  direction  peripheral  vision  sensor  PC-104  size  circuit  board,  (center  left)  Schematic  of  head  worn  cap 
with  embedded  IR  emitters  and  receivers  detecting  (center  right)  objects  in  the  environment,  (left)  A 
second  circuit  board  receives  the  signal  and  represents  the  information  on  the  torso  using  the  TTI. 


These  technology  advances  formed  the  basis  of  the  ACMI-VAS  central/paracentral  and 
peripheral  vision  substitution  concept.  Because  the  VI 00  is  now  available  as  an  investigational 
device  for  research  applications,  IHMC  designed  a  study  to  compare  blind  performance  on 
activities  of  daily  living  with  the  BrainPort®  alone,  or  with  ACMI-VAS  (which  includes  the 
BrainPort®  for  foveal  substitution,  the  VideoTact  for  parafoveal  substitution,  and  the  TTI  for 
peripheral  substitution). 

In  the  first  project  quarter,  primary  efforts  related  to  initial  evaluation  of  hardware 
integration  approaches  for  development  of  the  multisensory  “haptic  retina”  system.  This 
included  evaluation  and  modification  of  existing  training  protocols  by  the  research  staff  and 
software  modification  to  parse  data  from  a  single  camera  to  multiple  tactile  displays  at 
resolutions  appropriate  for  each  display.  We  integrated  the  current  VideoTact  display  with  older 
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prototype  325  pixel  BrainPort®  devices.  Two  hardware  issues  arose  in  the  first  quarter  that 
delayed  initial  integration.  The  first  related  to  this  VideoTact  (ForeThought  Development,  LLC, 
Blue  Mounds,  Wl)  electrotactile  abdominal  display,  which  developed  an  intermittent  issue  that 
resulted  in  a  system  shutdown.  This  resulted  from  a  safety  shutdown  whenever  a  signal  was 
passed  through  a  failed  resistor  array.  Repair  of  this  intermittent  failure  took  approximately  8 
weeks  from  onset  to  return  of  the  device.  The  second  issue  related  to  the  BrainPort®  VI 00 
electrotactile  tongue  displays  (Wicab,  Inc.,  Madison,  Wl)  that  were  delayed  by  4  weeks  due  to  a 
part  supply  issue  and  then  damaged  in  shipping  which  required  return  to  the  manufacturer.  Prior 
to  the  VideoTact  unit  failure,  we  were  able  to  determine  that  the  research  staff  could  maintain 
SA  during  of  a  multiple  line  shape  recognition  visual  task  more  effectively  when  using  the 
VideoTact  at  a  zoom  level  of  75  percent  of  that  presented  on  the  BrainPort®.  A  design  error  was 
detected  in  a  digital  interface  board  related  to  monitoring  of  the  status  of  the  vibrotactile  TTI  that 
did  not  affect  function,  but  that  required  fabrication  of  two  small  replacement  circuit  boards  to 
avoid  excessive  software  manipulation  to  maintain  proper  mapping  with  the  peripheral  vision 
sensor.  We  purchased  and  assembled  a  number  of  visual  task  items  to  evaluate  the  potential 
for  testing  color  perception  with  blind  individuals. 

The  second  quarter  efforts  included  evaluation  of  modified  training  protocols  by  the 
research  staff,  hardware  modifications  to  simply  operations  during  the  protocol  software 
modification,  evaluation  of  optimal  relative  field  of  view  between  BrainPort®  and  VideoTact 
displays  for  parafoveal  functionality.  We  currently  have  integrated  the  current  VideoTact  signal 
with  our  600  pixel  BrainPort®  devices.  We  have  integrated  a  servo  controlled  color  wheel 
system  that  mounts  in  front  of  the  BrainPort®  camera  and  switches  between  red,  blue,  green 
and  neutral  (clear)  filters  under  user  control.  This  allows  a  blind  (or  blindfolded)  user  to 
determine  the  color  of  the  red,  green  and  blue  faces  of  an  illuminated  cube  or  button  pad.  We 
completed  fabrication  of  a  custom  frame  and  curtain  system  that  completely  blacks-out  the 
participant  testing  room  ensuring  high  contrast,  low  visual  noise  during  training.  Following 
extensive  mock  up  and  protocol  testing  with  research  staff  (see  figures  8-10),  we  completed  and 
submitted  an  application  for  use  of  human  research  participants  to  the  IHMC  institutional  review 
board  (IRB). 


Figure  7:  Research  staff  testing  Landolt  "C"  orientation  test  in  training  room  (draped  in  black  felt).  Staff 
member  is  using  the  BrainPort®  VI 00  device  and  wearing  the  TTI  now  embedded  in  an  Army  Combat 
Uniform  (ACU)  tactical  vest. 
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Figure  9:  (left)  Research  staff  (wearing  TTI  vest  and  using  BrainPort  )  identifying  a  golf  ball  from  a 
baseball,  (right)  Research  staff  examining  illuminated  multicolored  cube. 


Figure  10:  (left)  Research  staff  navigating  indoor  ambulation  course  while  avoiding  high  contrast  foam 
obstacles  suspended  from  ceiling,  (right)  Research  staff  navigating  a  different  part  of  the  course  with  floor 
level  obstacles. 

During  the  third  quarter,  the  research  staff  practiced  the  training  protocols  and  improved 
the  user  control  interface.  This  included  the  development  of  a  World  Wide  Web  navigation 
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simulation  for  the  computer  based  reading/navigation  portion  of  the  protocol.  This  allows  us  to 
simulate  the  interactions  participants  would  have  with  web-based  activities  (searching,  social 
networking,  etc.)  while  logging  mouse  activity,  time  to  find  specified  links  and  accuracy  when 
detecting  those  links.  We  have  completed  practice  runs  through  the  entire  protocol  and  verified 
the  data  collection  software  (Figure  11). 
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Figure  11:  Screenshots  from  simulated  web  navigation/reading  proficiency  task.  The  test  starts  with 
simple  tasks  such  as  identifying  a  word  and  advances  to  finding  the  link  for  a  friend  on  a  Facebook  page 
and  clicking  through  the  conversation.  The  software  calculates  the  number  correct  and  react  time  for  each 
task.  Rather  than  use  a  camera  to  image  the  monitor,  which  is  the  images  are  sent  directly  to  the  ACMI- 
VAS  system  from  the  computer. 

We  also  integrated  wireless  Playstation3  game  console  hand  controllers  (Sony,  Corp., 
Tokyo,  Japan)  into  the  ACMI-VAS  architecture.  These  video  game  interface  devices  should  be 
familiar  to  our  target  population  of  recently  blinded  servicemembers  and  reduce  familiarization 
time.  We  completed  fabrication  of  the  servo  controlled  red/green/blue/clear  color  camera  filter 
for  the  color  perception  component  of  the  protocol  in  ABS  plastic  using  IHMC’s  3D  printer 
(Figure  12). 


Figure  12:  Computer  aided  design  model  of  removable  servo  controlled  camera  color  filter  and  active 
color  sensor. 

The  color  filter  was  mounted  to  an  unmodified  BrainPort®  V-100  vision  device  that  has  a  400 
pixel  intraoral  display  (IOD)  and  can  also  be  mounted  to  an  earlier  BrainPort®  Wearable  Aid  for 
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Vision  Enhancement  (BP-WAVE-II)  with  a  600  pixel  IOD.  Additionally,  we  implemented  an 
active  color  sensor  (ColorPAL,  Parallax,  Inc.,  Rocklin,  CA)  to  the  filter  system  that  provides 
controlled  color  illumination  for  measurement  of  actual  red-green-blue  (RGB)  values  for  use  with 
objects  that  are  not  self-illuminated.  During  initial  evaluation  of  this  system,  the  research  staff 
could  easily  recognize  primary  colors,  as  well  as  secondary  colors,  presented  on  a  4x4  array  of 
RGB  light  emitting  diodes  (LEDs)  mounted  under  translucent  white  momentary  pushbuttons 
(Buttonpad  ControllerUSB,  Sparkfun  Electronics,  Boulder,  CO).  Users  can,  with  a  slight 
increase  in  effort,  accurately  describe  color  variations  (e.g.,  reddish  orange)  (Figure  13). 
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Figure  13:  (left  6  images)  Research  staff  using  BrainPort  fitted  with  user  controlled  RGB  filter  (white  disk 
mounted  to  glasses)  to  identify  the  color  presented  on  the  LED  Buttonpad.  The  "D-Pad"  buttons  on  the 
Playstation3  game  controller  allow  the  user  to  switch  between  red,  green,  blue  and  clear  filters,  (right) 
Integrated  hand  controller  and  software  (foreground)  for  color  identification  task. 


In  the  fourth  quarter,  we  received  approval  for  use  of  human  research  participants  for  the 
ACMI-VAS  protocol  from  the  IHMC  IRB.  The  approval  letter  was  forwarded  to  the  U.  S.  Army 
Human  Research  Protection  Office  (HRPO),  which  then  requested  a  number  of  clarifications 
and  modifications  to  the  previously  submitted  HRPO  application  packet.  We  submitted  the 
requested  information  as  well  as  additional  information  as  requested  by  the  next  level  of  review. 
We  are  currently  awaiting  HRPO  requests  for  further  clarifications  or  final  determination.  While 
awaiting  final  approval  to  being  subject  recruitment,  we  have  rehearsed  the  protocol  with  the 
research  staff  and  made  changes  and  updates  to  the  hardware  and  software  to  ensure  smooth, 
consistent  protocol  execution.  This  included  sewing  hook  and  loop  fasteners  to  the  indoor 
ambulation  course  runner  sections  to  remove  any  tripping  hazard  at  the  overlapping  runner 
joints  and  testing  placement  of  ambulation  course  obstacles.  We  also  verified  the  functionality  of 
the  simulated  web  browser  task  as  well  as  the  scoring  algorithms.  Lastly,  because  the  color 
identification  task  appears  trivial  for  primary  and  secondary  colors,  we  have  added  an  additional 
function  that  sets  one  LED  on  the  Buttonpad  array  to  a  different  color  than  the  other  fifteen 
LEDs.  We  have  determined  that  blindfolded  individuals  can  quickly  identify  the  background  color 
and  both  the  location  and  color  of  the  oddball  LED  accurately.  We  have  also  integrated  the 
Button  pad  and  a  Playstation3  controller  into  a  single  handheld  unit  that  simplifies  user 
interactions  with  the  system  (Figure  13,  right).  The  Playstation3  controller  portion  controls  the 
color  filter  selection,  the  ACMI-VAS  interfaces  (e.g.,  zoom  level,  intensity,  black/white  inversion, 
etc.)  while  the  button  pad  displays  the  colors  and  detects  responses  when  the  user  presses  the 
button  over  the  LED.  It  connects  to  the  control  computer  as  well  has  the  color  filter  via  universal 
serial  bus  (USB).  This  integration  of  the  Playstation3  controller  allows  the  use  to  manage  all  of 
the  ACMI-VAS  elements  with  a  single  hand  held  device.  For  the  ambulation  tasks,  which  do  not 
require  the  Button  pad  display,  a  standard  Playstation3  controller  connects  wirelessly  to  the 


Approved  for  public  release;  distribution  unlimited  15 


control  computer  via  Bluetooth™.  The  bulk  of  the  effort  on  the  project  this  quarter  focused  on 
preparing,  submitting  and  revising  the  IRB  applications  to  the  IHMC  IRB  and  the  Army  HRPO. 

Relationship  to  award  Statement  of  Work 

At  the  end  of  the  first  year  of  the  program,  we  have  developed  a  multisensory  interface 
that  provides  visual  substitution  via  tactile  displays  that  provides  a  partial  functional  restoration 
of  visual  perception.  Based  on  the  original  statement  of  work,  we  have  completed  the  following 
tasks: 

Specific  Aim  #  1:  Develop  a  multi-sensory  tactile  interface  to  augment  or  substitute  for  recently 
acquired  visual  impairment 

Task  1.  Integrate  IHMC  Tactile  interfaces  for  haptic  retina  application 

la.  Finalize  sensor  integration  plan  for  peripheral  and  central  vision  functions 

lb.  Develop  integrated  abdomen,  torso  and  tongue  display  mounting  system 

lc.  Develop  multiple  sensor  hardware  mounting  system 

ld.  Develop  software  functional  requirements 
Milestone  #1_j  ACMI-VAS  cntjcal  design  review 

Task  2.  Integrate  visual  sensory  augmentation/substitution 

2a.  Evaluate  feasibility  of  providing  stereoscopic  vision  with  current  interfaces 
2b.  Define  paracentral  display  icon  for  registration  of  foveation  point 

Specific  Aim  #2:  Evaluate  ACMI-VAS 

Task  3.  Code  software  for  visual  environment  interactions 

3a.  Integrate  central  (foveal),  paracentral  and  peripheral  vision  manual  controls 

3b.  Develop  calibration  procedures  for  ACMI-VAS 

3c.  Code  user  interface 

3d.  Code  performance  evaluation  interface 

3e.  Test  and  evaluate  interface  function  with  tactile  and  audio  displays 
3f.  Define  performance  metrics  and  code  software  for  evaluation  of  metrics 
3g.  Verify  evaluation  software  functionality 

Milestone  #3:  ACMI-VAS  haptic  retina  evaluation  system  design  complete 

Task  4.  Demonstration  and  evaluation 

4a.  Submit  IRB  and  HRPO  applications  for  use  of  human  research  participants 
4b.  Prototype  system  functional  verification  and  testing 

Remaining  Tasks/Milestones: 

Task  2.  Integrate  visual  sensory  augmentation/substitution 
2c.  Fabricate  hardware  for  IHMC  peripheral  vision  sensor 
2d.  Integrate  software  (using  AMI)  of  IHMC  peripheral  vision  sensor 
2e.  Test  &  evaluate  ACMI-VAS  haptic  retina  software/hardware  integration 
Milestone  #2:  Complete  ACMI-VAS  haptic  retina  prototype  system 

Task  4.  Demonstration  and  evaluation 

4c.  Identify  and  recruit  20  recently  blinded  research  participants 

4d.  Human  participant  testing 

4e.  Human  participant  data  analysis 

Milestone  #4:  Collect  and_  analyze  ACMfVAS  human  research  participant  data 
Milestone  #5:  Draft  manuscript  for  submission  to  Ophthalmology  (journal} 
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Deliverables:  1)  HRPO  application  for  use  of  human  research  participants,  2)  Quarterly  reports, 
and  3)  Annual  Report 

Key  research  accomplishments 

•  We  successfully  integrated  a  prototype  haptic  retina  via  the  AMI  software  architecture 

•  We  determined  that  serial  visual  tasks  such  as  reading  and  identifying  shapes  on  a  grid  was 
enhanced  when  using  ACMI-VAS 

•  We  determined  that  colors  could  be  identified  tactually. 

•  We  developed  a  test  protocol  to  evaluate  the  effectiveness  of  the  ACMI-VAS  concept 
against  the  BrainPort®  VI 00  alone. 

Reportable  outcomes 

None  to  date,  awaiting  HRPO  approval  to  begin  human  participant  testing. 

Conclusion 

During  year  one  of  the  ACMI-VAS  project,  we  developed  a  prototype  ACMI-VAS  system 
and  the  research  environment  needed  to  evaluate  it.  We  improved  the  user  control  interfaces 
and  developed  a  method  to  allow  tactual  understanding  of  color.  The  final  portion  of  this  grant 
will  focus  on  human  research  participant  testing  and  evaluation,  data  analysis,  drafting  a 
publication  detailing  the  results,  and  development  of  the  final  ACMI-VAS  prototype  design 
specification  document.  The  noninvasive  nature  of  the  ACMI  approach  ensures  that  injured 
servicemembers  could  benefit  from  future  upgrades  as  technologies  improve  (in  out-years) 
without  risks  of  further  surgeries  or  infection  associated  with  implantable  devices.  The  proposed 
complementary  interface  displays  can  be  tailored  to  suit  the  needs  of  an  individual.  For 
example,  an  injury  that  spared  the  peripheral  vision  may  only  require  the  higher  resolution 
displays,  whereas  a  condition  like  hemianopsia  might  only  require  a  low-resolution  spatial 
awareness  component.  This  proposed  technology  development  will  result  in  a  single  integrated 
system  prototype  capable  of  providing  an  alternative  mechanism  for  visual  sensing  of  high 
resolution  foveal  vision,  low  resolution  peripheral  vision  and  stabilization  of  the  imagery  despite 
perturbations  of  the  head.  Even  profoundly  blind  individuals  would  benefit  from  the  modularity  of 
the  system  as  they  could  choose  to  use  specific  displays  for  any  given  activity.  The  use  of  the 
AMI  software  agent  framework  ensures  that  integration  of  improvements  in  any  of  the  major 
technologies,  including  sensing  devices  (e.g.,  cameras)  and  interfaces  (potentially  even 
implantable  ones)  will  occur  quickly,  speeding  up  evaluation  of  incremental  changes  and  their 
deployment  to  the  users. 
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Appendix:  Acronyms 


•  ACMI-VAS  -  Anthro-Centric  Multisensory  Interface  for  Vision  Augmentation/Substitution 

•  AMI  -  Adaptive  Multiagent  Integration 

•  BP-WAVE  II-  BrainPort®  Wearable  Aid  for  Vision  Enhancement,  version  II 

•  HRPO  -  Human  Research  Protection  Office 

•  IHMC  -  Florida  Institute  for  Human  and  Machine  Cognition 

•  IOD  -  Intra  Oral  Device 

•  IRB  -  Institutional  Review  Board 

•  JAWS  -  Job  Access  With  Speech 

•  LED  -  Light  Emitting  Diode 

•  NASA  -  National  Aeronautics  and  Space  Administration 

•  RGB  -  Red,  Green,  Blue 

•  SA  -  Situation  Awareness 

•  TBI  -  Traumatic  Brain  Injury 

•  TSAS  -  Tactile  Situation  Awareness  System 

•  TTI  -  Tactile  Torso  Interface 

•  USB  -  Universal  Serial  Bus 
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